Skip to content

ref(consumer): remove the BLQ (backlog-queue) mechanism - #8166

Merged
onewland merged 1 commit into
masterfrom
ref/remove-blq
Jul 16, 2026
Merged

ref(consumer): remove the BLQ (backlog-queue) mechanism#8166
onewland merged 1 commit into
masterfrom
ref/remove-blq

Conversation

@onewland

@onewland onewland commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Removes the backlog-queue (BLQ) mechanism from the Rust consumer. BLQRouter redirected stale messages (by Kafka broker timestamp) to the DLQ topic and panicked the consumer to flush in-memory state on the fresh→stale transition. It was never fully rolled out, the FSM (Idle/RoutingStale/Flushing/Forwarding + panic-to-reset) was hard to reason about, and no storage enabled it in prod.

This is a pure removal (−481 lines, no additions). The DLQ policy/topic are untouched.

Removed

  • rust_snuba/src/strategies/blq_router.rs (the strategy + its unit tests)
  • pub mod blq_router; in strategies/mod.rs
  • BLQ import, the blq_producer_config/blq_topic fields, and the pipeline-wrapping block in factory_v2.rs
  • The blq_producer_config builder + factory args in consumer.rs
  • The two blq_*: None bench fields
  • The 3 consumer.blq_* options from the sentry-options schema

Why

Clears the way for a simpler DLQ-by-age lever (follow-up PR) built on the standard InvalidMessage mechanism, so we can shed a configurable proportion of too-old messages to the DLQ during a backlog and backfill later.

Test plan

  • cargo check --all-targets — clean, no warnings
  • cargo test --lib strategies — 36 pass

🤖 Generated with Claude Code

The backlog-queue router (BLQRouter) redirected stale messages, by Kafka
timestamp, to the DLQ topic and panicked the consumer to flush in-memory
state on the fresh->stale transition. It was never fully rolled out, the
FSM (Idle/RoutingStale/Flushing/Forwarding + panic-to-reset) was hard to
reason about, and no storage enabled it in prod.

Remove the strategy, its wiring in factory_v2/consumer, the bench fields,
and the consumer.blq_* sentry-options. The DLQ policy/topic are untouched.

This clears the way for a simpler DLQ-by-age lever built on the standard
InvalidMessage mechanism.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@onewland
onewland marked this pull request as ready for review July 8, 2026 19:43
@onewland
onewland requested a review from a team as a code owner July 8, 2026 19:43
@onewland
onewland merged commit 45bb780 into master Jul 16, 2026
70 checks passed
@onewland
onewland deleted the ref/remove-blq branch July 16, 2026 22:21
onewland added a commit that referenced this pull request Jul 17, 2026
> **Stacked on #8166** (BLQ removal). Base is `ref/remove-blq`; review
that first. Retarget to `master` once #8166 merges.

## Summary

Adds `DlqByAge`, a stateless arroyo strategy for the Rust consumer that
sheds load during a backlog. When enabled, it routes a **configurable
proportion** of messages older than a threshold (by **Kafka broker
timestamp**) to the DLQ via the standard `InvalidMessage` mechanism. The
rest are forwarded untouched — so a backlogged consumer catches up to
fresh data while the parked backlog is backfilled from the DLQ later.

This replaces the removed BLQ mechanism with something far simpler: **no
state machine, no consumer panic, no separate producer.** It leans
entirely on arroyo's existing DLQ buffer + policy.

## How it works

- Sits at the front of the pipeline (where `BLQRouter` was), *before*
decode/process, so dropped messages cost almost nothing.
- For each message: if the per-storage sample rate is 0 → forward (fast
path). Else if `now - broker_ts <= threshold` → forward. Else roll `rand
< rate`: hit → `Err(SubmitError::InvalidMessage)` (→ DLQ); miss →
forward.
- **Only inserted when a DLQ topic is configured** (`dlq_configured`
flag from `consumer.rs`). Without a DLQ policy, arroyo logs an
`InvalidMessage` and silently drops it — which would lose data — so we
never wire the strategy in that case.

## Kept out of Sentry

Age-based routing is an expected load-shedding outcome, not an error.
Dead-lettered messages use `InvalidMessageReason::Ignored`, so arroyo
logs them at DEBUG (not ERROR) and the tracing→Sentry layer
(`logging.rs`) does **not** turn them into Sentry issues. Both reasons
produce to the DLQ identically — only the log level differs. This is the
same intent as the `SilencedDLQMessage` path from #8012.

## Configuration (sentry-options, read per-message, runtime-tunable)

| Option | Type | Default | Meaning |
|---|---|---|---|
| `consumer.dlq_by_age_sample_rate_by_storage` | dict `{storage:
number}` | `{}` | fraction (0–1) of too-old messages to dead-letter; no
entry ⇒ 0 (disabled) |
| `consumer.dlq_by_age_threshold_seconds` | int | `3600` | age cutoff by
broker timestamp |

**Ships disabled** — no storage is enabled and the rate defaults to 0,
so this is an inert passthrough until an operator opts a storage in.

Metrics: `dlq_by_age.stale_seen` and `dlq_by_age.routed_to_dlq` (storage
is already a global metric tag).

## Test plan
- `cargo test --lib dlq_by_age` — 6 unit tests pass:
disabled-by-default, explicit rate 0, fresh-forwarded, rate 1 routes all
stale, per-storage scoping, and a partial-rate split. Each DLQ'd message
is asserted to carry `Ignored`.
- `cargo clippy --all-targets` clean; `cargo fmt --check` clean.

## Follow-up (PR 3, ops)
Enable per storage via sentry-options and watch
`dlq_by_age.routed_to_dlq` + DLQ topic lag.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants